@elsif2 the license check is failing

From the log:

The following files have no copyright and licensing information:
* intelmq/bots/parsers/shadowserver/schema.json.test

License added.

In addition, I'd like to ask if you can publish a policy on changes in the schema, the most important for me is in which cases you can change classification config (e.g. identifier, taxonomy)

I haven't seen a discussion of this architectural change or and IEP yet. Or was it discussed on a non-public list?

In addition, I'd like to ask if you can publish a policy on changes in the schema, the most important for me is in which cases you can change classification config (e.g. identifier, taxonomy)

I started a Schema contract section to address concerns in the bots/parsers/shadowserver/README.md.

I haven't seen a discussion of this architectural change or and IEP yet. Or was it discussed on a non-public list?

@sebix I had a good chat with elsif2 today and we said

• I try out the changes, get it to work for myself
• document things as I go
• cross check with elsif2 as I got and we change things iteratively
• we both start to write down sort of a "contract" / documentation on this . It's paramount to know for intelmq-users and -admins what the "contract" is: what fields may change dynamically, added fields will stay for ever, only new fields get added, how to deal with new fields? How often updates are needed? What happens when someone does not update? ETc. etc... If we write this down and make it clear, it is the basis for discussions and it helps also to improve the concept.

So, TL;DR: don't worry :) we are adressing concerns.

I haven't seen a discussion of this architectural change or and IEP yet. Or was it discussed on a non-public list?

* we both start to write down sort of a "contract" / documentation on this . It's paramount to know for intelmq-users and -admins what the "contract" is: what fields may change dynamically, added fields will stay for ever, only new fields get added, how to deal with new fields? How often updates are needed? What happens when someone does not update? ETc. etc... If we write this down and make it clear, it is the basis for discussions and it helps also to improve the concept.

This is a good step forward!

So, TL;DR: don't worry :) we are adressing concerns.

I'm not worried personally, as I'm not directly affected, but many users are. I want to contribute my experience to get the change into operation smoothly.

Needless to say, the proposed approach has its advantages.

Here are some thoughts:

• The delegation of the mapping to a third party outside of the IntelMQ software also deprives the IntelMQ developers & users of the possibility of reviewing the field mappings before they become active.
• Until now, only collectors fetch the Shadowserver data via API or from local sources like IMAP, file, ticket system, etc. IntelMQ doesn't need to be connected to the internet to parse Shadowserver data. With this change merged, IntelMQ gains an additional requirement on the operation. Operators need to consider that and eventually adapt their installations or environments.
• Another issue is that it will take more work to parse older or historical data. Do you remember that a few years ago, we parsed Shadowserver data covering a few years of data for statistical purposes? We had the older Shadowserver feed mappings at hand, thus no issues parsing older data.

Let me try to address some topics mentioned.

• The delegation of the mapping to a third party outside of the IntelMQ software also deprives the IntelMQ developers & users of the possibility of reviewing the field mappings before they become active.

nope, since you can always fetch the newest mappings manually and review. So, I don't see a problem here.
But yes, we could and should try out the "air-gapped" mode and see if it's actually feasible before merging. Agreed with that.

• Until now, only collectors fetch the Shadowserver data via API or from local sources like IMAP, file, ticket system, etc. IntelMQ doesn't need to be connected to the internet to parse Shadowserver data. With this change merged, IntelMQ gains an additional requirement on the operation. Operators need to consider that and eventually adapt their installations or environments.

Again see above, it will also work in airgapped mode and pulling the latest mappings will be similar to doing an intelmq update manually.

• Another issue is that it will take more work to parse older or historical data. Do you remember that a few years ago, we parsed Shadowserver data covering a few years of data for statistical purposes? We had the older Shadowserver feed mappings at hand, thus no issues parsing older data.

This is a very relevant point. I think this needs to be in the contract that historic fields don't get remapped or changed or deleted.

Thanks for changes, @elsif2 I see most of my technical concerns have been addressed :)

About the general discussion:

re: delegation of schema outside IMQ

I see this as a trade-off problem, but I think ShadowServer has a very good reason to delegate it outside as their reports are developing much quicker than IntelMQ. In addition, I see a possibility for automated upgrades of this schema as a big facilitation for admins - even with a very quick adoption in IMQ packages, you still need to upgrade the whole IntelMQ instead of just the schema. Yes, it has some cons, but I think the pros are stronger in this case.

re: air-gaped systems

This is why I'd suggest keeping default, regularly updated (e.g. once a month) schema in the repo. Then your air-gaped system can still work after installation, just without additional configuration it would have a delay in adoption of schema changes.

re: historical schema

This is a good point. But I'd just suggest keeping the history of schemas, in to ways:

• do not delete old schema, create a history folder instead and store them all downloaded schemas, named after the schema version.
• For cases when you don't have the schema, @elsif2 would it be possible, that ShadowServer publish a public catalog with all historical schemas?

As I understand, the schema is versioned by time, so it should be easy to choose the right one if you need to perform a reparsing of old reports - assuming that ShadowServer would ensure that date of creation a schema means, that starting this or the next day the schema is used (please clarify it in the contract)

The complete revision history of the schema files will be available at https://github.com/The-Shadowserver-Foundation/report_schema.

Thank you!

looks like my requests have been met.
Testing again now.

I wanted to cancel the pending review for myself requested by Aaron, but it looks like I instead canceled one of Aaron's reviews 🤔

Anyway, I won't review this PR again, as I already reviewed some parts of it (core, docs). The major and more relevant parts have already been reviewed by Kamil and Aaron. I don't want to duplicate their efforts. I shared my thoughts on the change in general already above (#2372 (comment))

@sebix all good, I am still looking at it. It's a mega mega mega commit. Need to do a smaller one the next time to make the review process faster. Most of us have other duties / daytime jobs etc.

This branch is currently in production usage on our instance, so far I do not see any issues.

We would really like to see this feature merged in IntelMQ. We know the pain of manually merging the _config.py whenever new reports are added too well. The changes made to host(name) on the 18th of November showed the problem where we would require a different config from one day to another. The dynamic config would fix those kinds of issues as well.

We do however see a problem. We run a custom config with a couple of changes compared to the upstream config:

• We changed the classification.identifer of scan_http_vulnerable from accessible-http to vulnerable-http. accessible-http is also used by scan_http, which is a completely different report.
• event_honeypot_brute_force, event_honeypot_darknet, event_sinkhole_http and event_sinkhole map fields like tag to classification.identifer. We would like a constant value for classification.identifer, so we moved it to the constant_fields section with values honeypot-brute-force, darknet, sinkhole-http and sinkhole.
• sandbox_url and sandbox_conn (unnecessary in our opinion) mix source and destination fields. We changed destination.fqdn and destination.to source.fqdn and source.url.

We would not be surprised if other organizations have their own changes that they made to the config.
A solution would be to allow us to configure a path to a JSON patch document (RFC 6902) in the config. This patch is applied to the schema JSON after a new version is downloaded in the update process.

Hey, we indeed have a couple of custom changes (e.g. filters by tags, own identifiers etc.), but we use a sieve bot for that - and as long as the feed.code is stable, our bot handles that independently. I would recommend this approach instead of modifying ´_config.py` or patching schema

Thanks for the pointer. We are now looking into improving our setup using the sieve bot. The only thing it cannot solve is the change of the fields for sandbox_url (you cannot add a new field with a value from another field). Maybe those fields should be changed in the upstream schema itself. However, that is not relevant for this PR.

The change occurred between 3.0.2 and 3.1.0. I will correct this in the schema.

3.0.2

sandbox_url = {
    'required_fields': [
        ('time.source', 'timestamp', add_UTC_to_timestamp),
        ('source.ip', 'ip'),
    ],
    'optional_fields': [
        ('source.asn', 'asn', invalidate_zero),
        ('source.geolocation.cc', 'geo'),
        ('malware.hash.md5', 'md5hash'),
        ('source.url', 'url'),
        ('user_agent', 'user_agent', validate_to_none),
        ('source.fqdn', 'host', validate_fqdn),
        ('extra.', 'method', validate_to_none),
    ],
    'constant_fields': {
        'classification.taxonomy': 'malicious-code',
        'classification.type': 'malware-distribution',
        'classification.identifier': 'sandbox-url',
    },
}

3.1.0

sandbox_url = {
    'required_fields': [
        ('time.source', 'timestamp', add_UTC_to_timestamp),
        ('source.ip', 'ip', validate_ip),
    ],
    'optional_fields': [
        ('destination.fqdn', 'host', validate_fqdn),
        ('extra.http_request_method', 'method', validate_to_none),
        ('source.asn', 'asn', invalidate_zero),
        ('source.geolocation.cc', 'geo'),
        ('malware.hash.md5', 'md5', validate_to_none),
        ('destination.url', 'url', convert_http_host_and_url, True),
        ('user_agent', 'user_agent', validate_to_none),
    ],
    'constant_fields': {
        'classification.taxonomy': 'malicious-code',
        'classification.type': 'malware-distribution',
        'classification.identifier': 'sandbox-url',
    },
}

sandbox_conn = {
    'required_fields': [
        ('time.source', 'timestamp', add_UTC_to_timestamp),
        ('source.ip', 'ip', validate_ip),
        ('source.port', 'port', convert_int),
    ],
    'optional_fields': [
        ('destination.fqdn', 'host', validate_fqdn),
        ('source.asn', 'asn', invalidate_zero),
        ('source.geolocation.cc', 'geo'),
        ('malware.hash.md5', 'md5', validate_to_none),
        ('protocol.transport', 'protocol'),
        ('extra.', 'bytes_in', validate_to_none),
        ('extra.', 'bytes_out', validate_to_none),
    ],
    'constant_fields': {
        'classification.taxonomy': 'malicious-code',
        'classification.type': 'malware-distribution',
        'classification.identifier': 'sandbox-conn',
    },
}

Thanks for the pointer. We are now looking into improving our setup using the sieve bot. The only thing it cannot solve is the change of the fields for sandbox_url (you cannot add a new field with a value from another field).

That is true. As a workaround, you could use the Jinja2 Template Expert or the Modify Expert

And all checks have passed :) merging in...